Design a semantic search system that must support 500 million documents with sub-100ms p99 latency. What are the key architectural decisions?

The first decision is storage layout, because 500M documents at 768 dimensions of float32 is roughly 1.5 TB of raw vectors before any index. That will not fit in a single node's RAM at reasonable cost, and even if it did, the HNSW graph would add another 80-160 GB on top. The right answer is a hybrid: keep quantized vectors resident in RAM for traversal, keep the full-precision vectors on disk for rescoring, and put the HNSW graph on disk with inline quantized vectors to halve the I/O per traversal step. This combination makes sub-100ms p99 achievable without a huge RAM bill. The second decision is sharding: split the collection across enough shards that each shard fits comfortably on a node and each node can serve its share of the QPS. For 500M documents, 8-16 shards per replica set is a reasonable starting point, but the exact count depends on the query pattern and whether queries can be routed to a single shard via a shard key. The third decision is replication: for a mission-critical system, replication_factor=2 or 3 to survive a node failure without an outage. The fourth decision is caching: embedding cache at the application layer, result cache for repeated queries, and the OS page cache for the hot portion of the on-disk data. The fifth decision is the query path: use a two-stage pipeline with a cheap first-stage retriever and a reranker over a small candidate set.

The mechanism that lets this hit sub-100ms p99 is that each stage has a bounded cost. Quantized ANN traversal touches only the small quantized vectors, so it is memory-bandwidth-bound rather than disk-bound, and the graph is traversed in microseconds per hop. Rescoring touches only the top ~100 candidates with full-precision vectors, so the disk I/O is bounded and mostly cached. Sharding parallelizes the traversal across nodes, so the per-shard latency is a fraction of the total work, and the coordinator merges the per-shard top-k. Replication lets reads be distributed across replicas, which increases effective throughput and reduces the per-replica load. The p99 is dominated by the slowest shard and by the tail of the page-cache miss distribution, so the design must ensure that the working set fits in cache and that no single shard is a hotspot. The common failure mode is to underestimate the graph overhead and the payload index footprint, which pushes the working set out of cache and causes p99 spikes. Another common failure is to fan out queries to every shard on every request, which makes the coordinator and the slowest shard the bottleneck.

The trade-offs are between recall and latency (quantization and ef), cost and latency (RAM vs disk), and availability and cost (replication). The common mistakes are: (1) sizing RAM by raw vector bytes only, forgetting the graph, payload indexes, and quantized vectors; (2) not planning the shard count generously up front, since sharding is a heavier operation than adding replicas; (3) forgetting that the p99 is bounded by the slowest shard, so a single hot shard can dominate the tail; (4) not testing the whole pipeline end-to-end under realistic load, so the benchmark is optimistic; (5) assuming that more shards always means faster, when at high shard counts the coordinator merge cost dominates. Version note: on-disk HNSW, inline storage, and the newer quantization schemes are recent additions to Qdrant and have evolved across releases. Verify the availability of these features on your version and benchmark the configuration with your data before committing to a design.

Version-dependent: the on_disk flags, inline storage, binary quantization, and the optimizer thresholds that control disk placement have all changed across Qdrant releases. The exact availability of these features and the defaults differ. Benchmark the design on your version with your data and your query distribution before committing, and re-benchmark after upgrades.